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The image of an object can vary dramatically depending on lighting, specularities/ reflections and shad- 
ows. It is often advantageous to separate these incidental variations from the intrinsic aspects of an image. 
Along these lines this paper describes a method for photographing objects behind glass and digitally 
removing the reflections off the glass leaving the image of the objects behind the glass intact. We describe 
the details of this method which employs simple optical techniques and independent components analysis 
(ICA) and show its efficacy with several examples. 



1 Introduction 

The image of an object can vary dramatically depend- 
ing on lighting, specularities/ reflections and shadows, 
and yet our recognition of objects is amazingly robust 
despite these incidental variations. Our visual system 
seems to separate the various components that con- 
tribute to the formation of an image, yielding stable and 
reliable percepts. One commonly occurring example is 
that of reflections from dielectric surfaces (e.g., glass) 
where our visual system seems to easily ignore reflec- 
tions when, for example, viewing a painting framed 
behind glass. To facilite such tasks as object recogni- 
tion, visual-based navigation, and scene segmentation 
we would like to design computer systems that have a 
similar ability to separate the incidental from the intrin- 
sic aspects of an image. Along these lines, this paper 
describes a method for photographing objects viewed 
through glass and digitally removing the reflections off 
the glass leaving the image of the objects behind the 
glass intact. Although we will concentrate on this par- 
ticular problem we believe that the general methods 
used may be useful for a broader class of image separa- 
tion problems. 

Light reflected off glass at an oblique angle is par- 
tially polarized suggesting that the strength of the re- 
flection can be manipulated with a linear polarizer. The 
reflection can however only be completely eliminated 
when the viewing angle to the glass is at the Brewster's 
angle, typically a severe angle resulting in significant 
geometric distortions [1]. Consider the pair of images 
in Figure 1 of Renoir's On the Terrace framed behind 
glass with a reflection of a mannequin ("Sheila"). These 
images were photographed through a linear polarizer 
oriented to maximize the reflection (left) and minimize 
the reflection (right). The camera was oriented approx- 
imately 30°from frontal parallel. Note that even at the 
minimal orientation the reflection is quite salient. 

There is some history in the computer vision commu- 
nity of trying to remove specular reflections from im- 
ages. In these cases, researchers concerned themselves 
with non-planar reflective surfaces where the reflection 
is highly localized and poses a problem for various com- 
puter vision algorithms such as stereo and motion esti- 
mation. These approaches fall into one of several gen- 
eral categories: imposing a Lambertian assumption [2], 
color-based [3, 4, 5], polarization-based [6, 7] and com- 
binations thereof [8]. Here we concern ourselves with 
the problem of removing reflections from a planar sur- 
face and take a completely different computational ap- 
proach than previously suggested. Another important 
distinction is that in removing the reflections off the 
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Figure 1: Renoir's On the Terrace with a reflection 
of Sheila photographed through a linear polarizer at 
orthogonal orientations, maximizing and minimizing 
the reflection. 



glass we leave the image of objects behind the glass 
intact. 

Let's first look more closely at the simple physics of 
reflections from planar surfaces. Shown in Figure 2 is an 
idealized example of a photograph of a painting behind 
a plate of glass. The final image is a linear combination 
of the light that is reflected by the painting and the light 
that is directly reflected by the glass. l The reflection 
from the glass will itself be an image of the scene in 
which the painting is viewed. The amount of light at a 
single point in the image can be expressed as: 



2/1 



aP + bR, 



(1) 



where P and R are the amount of light contributed 
by the painting and reflection, and a and b are multi- 
plicative constants. We would like to remove the con- 
tribution of the reflection R from the image y\, but the 
above equation only provides a single constraint in four 
unknowns. Additional constraints may be added by 
exploiting the fact that reflections are partially polar- 
ized and that a linear polarizer can be used to adjust 
the relative strength of the reflection. With respect to 
Equation (1), the linear polarizer has the effect of ma- 
nipulating the relative contributions of the painting and 
reflection. The second image takes the form: 



J/2 



cP + dR. 



(2) 



The above equation provides another constraint, but 
two new unknowns have also been introduced, leav- 
ing us with a total of two constraints in six unknowns, 
and little hope of a solution without making further as- 
sumptions. It would of course be possible to manually 



1 In actuality, light is reflected off both the front and back face of 
the glass. In our conditions the pair of reflected images are nearly 
aligned so we assume the simplified model of an infinitely thin piece 
of glass with a single reflection. 
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Figure 2: A photograph of a painting behind glass 
contains a superposition of the light that is reflected 
by the painting, and the light that is reflected directly 
off the glass. 



find the amount of image y 2 that needs to be subtracted 
from y\ to remove the reflection R, but we are interested 
in an automatic procedure for accomplishing this. 

In order to separate the reflection from the desired 
image we first photograph a scene through a linear po- 
larizer at two distinct orientations. To solve the un- 
derconstrained set of equations we make the modest 
assumption that the image of the painting and reflec- 
tion are independent. Intuitively this means that for 
each spatial position the pixel intensity in one image 
provides no predictive information about the pixel in- 
tensity in the second image. This is a reasonable as- 
sumption since there is no reason to expect a correlation 
between the image of objects behind the glass and the 
image of objects reflected by the glass. Given the linear 
model of image formation and this assumption of inde- 
pendence we perform independent components anal- 
ysis (ICA) (e.g, [9, 10]) to separate the reflection from 
the desired image. In the next section the details of this 
statistical technique are outlined, and in the following 
section several examples of its efficacy are given. 



2 Separating Images 

The general problem of image separation can be stated 
as follows: given N distinct linear combinations of N 
images determine the original N images. For our ap- 
plication we can restrict ourselves to the case of just 
two images. Denoting these images in row vector form 
as x\ and x 2l the linear mixing of these images can be 
expressed in matrix form as follows: 
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Figure 3: Shown at the bottom left is an idealized joint 
probability distribution for a pair of independent im- 
ages. The linear mixing of these images, Equation (3), 
transforms this distribution, via a rotation, scaling, 
and rotation, from a square into a parallelogram (left 
column). The goal of ICA is to transform this paral- 
lelogram back into a square, thus yielding the original 
independent images (right column). 



where the matrix M embodies the linear mixing. Note 
that with this model it is assumed that the linear mixing 
is uniform over the entire image. The mixed images 
in Y each contain a linear combination of the source 
images in X. Our job is to recover the sources images 
from the mixed images. Of course given the full rank 
(i.e., invertible) matrix M it would be trivial to estimate 
the source images by left multiplying the mixed images 
with the inverse mixing matrix: 



X 



M~ l Y 



(4) 



But we don't typically know the mixing matrix, so our 
job will be to estimate it from the mixed images only. 

Equation (3) provides two constraints in six unknowns 
and so cannot be solved without further assumptions. 
The first assumption we make is that the pair of source 
images are independent. Denoting X\ and X2 as the 
random variables from which the pixel intensities of 
source images x\ and X2 are drawn, this assumption 
can be expressed as P(Xi,X 2 ) = P{X 1 )-P{X 2 ) (i.e., the 
joint probability distribution is separable). Although 



the constraint is expressed in terms of these continuous 
random variables we will typically work with the his- 
tograms of sampled images that, because of the dense 
sampling, are good approximations to the continuous 
probability distributions. The second modest assump- 
tion is that the mixing matrix M is full rank. With 
these two assumptions the general estimation problem 
is known as independent components analysis (ICA) 
of which there is a large and varied literature. Early 
contributors include [11, 9, 10, 12, 13, 14, 15, 16]. We 
present an analytic version of ICA based on higher- 
order statistical moments most similar to that of [9]. We 
provide a different formulation based on maximizing a 
series of error functions and give a simple and intuitive 
geometric interpretation of these steps. 

We begin by attempting to gain some insight into 
the structure of the mixing matrix by decomposing it 
according to the singular value decomposition (SVD): 
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R1SR2 



(5) 



where R\ and R2 are orthonormal (rotation) matrices 
and S is a diagonal (scaling) matrix. Shown in Fig- 
ure 3 is a geometric interpretation of the effects of each 
of these matrices on the joint probability distribution 
of the source images X. According to our assump- 
tion of independence this distribution should be sepa- 
rable; for illustration purposes consider the case when 
the marginal distributions are uniform, then the joint 
distribution is a square. The mixing matrix rotates, 
scales, and again rotates this distribution transform- 
ing the square into a parallelogram. Notice that the 
mixed images Y are no longer independent. The es- 
timation of the independent source images reduces to 
determining how to transform the parallelogram back 
into a square. Or more generally, transforming the two- 
dimensional joint distribution into a separable product 
of one-dimensional distributions. As Figure 3 suggests 
this may be accomplished by applying the opposite ro- 
tations and scalings in reverse order. 

The first step in separating the images is to apply 
a rotation that aligns the long and short axis of the 
parallelogram with the primary axis (Figure 3). These 
axes are easily determined since they are the axes with 
the maximal /minimal variance. Assuming zero mean 
measurements, the variance at an arbitrary orientation 
is given by: 
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The axis of maximal variance is determined by finding 
the angle 9\ that maximizes this error function (Fig- 



ure 4). The axis of minimal variance will be orthogo- 
nal to this axis oriented at 9\ — ir/2. These axes corre- 
spond to the principle axis as determined by principle 
components analysis (PCA). Since this error function is 
quadratic in its unknown it can be maximized analyti- 
cally by differentiating with respect to 9\, setting equal 
to zero and solving (Appendix A), to yield: 
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where, because it takes on a particularly simple form, 
the solution is given in polar coordinates r{i) = y\ (i) 2 + 
yiii) 2 and </>(i) = tan _1 (j/2(0/j/i(0)- Then, the first ro- 
tation matrix in the separation is: 



Ri 



cos(6*i) sin(6*i) 
-sin(6*i) cos(6*i) 



(8) 



Following the first rotation, the now aligned par- 
allelogram needs to be transformed into a diamond 
(Figure 3). More precisely, the axes need to be inde- 
pendently scaled so that the variance is rotationally 
invariant. The scaling of each axis is determined by 
first computing the variance along the axis of maximal 
and minimal variance, i.e., the axis oriented at 9\ and 
9 1 - ir/2: 
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and then the scaling matrix is constructed by placing 
the inverse variances along the diagonal: 
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(11) 



Combined, the first rotation and scaling are equiva- 
lent PCA plus whitening. But notice from Figure 3 
that this is insufficient for separating the mixed images 
into their independent components (the mixed images 
are only decorrelated, a necessary but insufficient con- 
dition). That is, in this example the joint probability 
distribution is in the shape of a diamond (i.e., is not 
separable), a final rotation is required to transform this 
diamond into a square, yielding the independent com- 
ponents (i.e., separability). 

One approach to the determination of this final ro- 
tation is to find the orientation #2 that maximizes the 
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Figure 4: Shown is the variation in the second mo- 
ment (variance) as the joint probability distribution is 
rotated and projected onto the horizontal axis (Equa- 
tion (6)). The variance is minimal along the short axis 
of the parallelogram and maximal at the orthogonal 
orientation. Note that this error function is a 2-cycle 
sinusoid. 



Figure 5: Shown is the variation in the fourth moment 
as the joint probability distribution is rotated and pro- 
jected onto the horizontal axis (Equation (13)). The 
fourth moment is maximal at the orientation required 
to transform the diamond into a square, which in turn 
yields the independent images. Note that this error 
function is a 4-cycle sinusoid. 



fourth statistical moment (Figure 5). The fourth mo- 
ment at an arbitrary orientation is given by: 
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where y[ and y' 2 are the result of rotating and scaling the 
initial mixed images y\ and y 2 according to Equations (8) 
and (11). Note that the above error function cannot 
be solved analytically. But, the following normalized 
fourth moment does lend itself to an analytic solution: 



E'(e 2 ) 
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(13) 



The axis where the fourth moment is maximal is deter- 
mined by finding the angle 9 2 that maximizes this error 
function (Figure 5). As before, we differentiate with 
respect to 6 2 , set equal to zero and solve (Appendix B) 
yielding the maximal solution: 
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again, for convenience, expressed in polar coordinates. 
The final rotation matrix then takes the form: 
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The estimation of the source images X from the 
mixed images Y is now a simple matter of applying 
the three matrices in Equations (8), (11), and (15): 
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(R 2 SRi)Y 



(16) 



There are two inherent ambiguities in the recovery 
of the independent components. First is the ordering 
ambiguity, that is, the following mixings are indistin- 
guishable: 



b a 
d c 



(17) 



Second is a scale ambiguity, that is the independent 
components can only be determined within a scale fac- 
tor since, for example, the following mixings are also 
indistinguishable. 



a b 
c d 
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For our purposes, the first of these ambiguities is not 
critical, and the second is dealt with by scaling the final 
images to fill the full intensity range. 

3 Results 

3.1 Synthetic 

Our first experiment is intended to show the general 
efficacy of the image separation outlined in the previ- 
ous section. Shown along the top row of Figure 6 are 
a pair of images and their normalized joint histogram 
(i.e., sampled joint probability distribution). In the next 
row are a pair of images formed by applying a random 
2x2 mixing matrix as in Equation (3) . In the subsequent 
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Figure 6: Along the top row are the original images 
of Einstein and Mandrill and their normalized joint 
histogram. The second row contains the mixed images 
and shown below are the intermediate steps leading 
to the separation of the mixed images. See also Table 1 . 
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Table 1: Results from the separation of Einstein and 
Mandrill, Figure 6. 
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Figure 7: Separating Einstein and Mandrill in the pres- 
ence of uniform zero-mean white noise. Shown is the 
root mean square (RMS) error between the separated 
and original images plotted as a function of the signal 
to noise ratio (SNR). 



three rows are the results of applying the rotations and 
scalings specified by Equations (8), (11), (15) and (16) to 
this pair of images. As desired, the mixed images have 
been separated into their original independent compo- 
nents. Note also the similarity in the transformations 
of the normalized joint histogram with the idealization 
of Figure 3. Shown in Table 1 are the initial and es- 
timated parameters of the mixing matrix. The pair of 
rotations are determined correctly within 1.7° and 6°, 
and the scaling within 3%. 

To measure the sensitivity to noise, varying amounts 
of uniform zero-mean noise were added to the mixed 
images y\ and j/2 before separating the independent 
components. Shown in Figure 7 is the root mean square 
(RMS) error between the separated and original images. 
The RMS errors are averaged over one hundred inde- 
pendent trials. 

3.2 Natural 

In the next set of experiments we photographed a paint- 
ing framed behind glass with the reflection of a man- 
nequin ("Sheila"), Figure 8. The painting with reflection 
was photographed twice through a linear polarizer ori- 
ented so that the reflection was maximized, and then at 
the orthogonal direction to minimize the reflection. We 
chose the maximum and minimum orientations to rein- 
force the fact that a linear polarizer alone is insufficient 
for removing the reflection. In general, any two dis- 
tinct orientations could be used that are not symmetric 
about the polarization angle of the reflected light. We 
used a color digital video camera (Canon Optura DV, 
Canon Inc.) whose video signal was digitized through 
a S-video connection onto a Silicon Graphics computer. 
The camera was calibrated to ensure a linear response. 
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Figure 8: Renoir's On the Terrace, Sheila and Sheila's 
reflection. 



Each scene was photographed for one-third of a second 
(10 frames) these frames were averaged in order to re- 
duce sensor noise. The final three channel (RGB) color 
images have a spatial resolution of 640 x 480 pixels. In 
our results the entire three channel image was used in 
computing the independent sources. 

Results are shown in Figures 9 and 10. Shown along 
the top row of each of these figures are the initial images 
and shown along the bottom row are the results of sepa- 
rating the independent components. In between are the 
intermediate steps leading to the separation as specified 
by Equations (8), (11), (15) and (16). Note how the nor- 
malized joint histogram (third column) transforms from 
a parallelogram to a square similar to the idealization 
of Figure 3, thus yielding the independent components 
- the painting and the reflection of Sheila. In the last 
experiment the reflections in the photograph of a store- 
front window are removed, shown in Figure 11 are the 
initial and separated images. 

4 Discussion 

We have developed a simple and effective method for 
separating reflections from images. This technique be- 
gins with a pair of images taken through a linear po- 
larizer at two distinct orientations. The reflection is 
separated from the image by applying an analytic ver- 
sion of independent components analysis (ICA) based 
on higher-order statistical moments. A real-time imple- 
mentation may be realized by synchronizing the image 
capture of alternate or even/odd frames with a liquid 
crystal polarizer (e.g, [17]). 

There are of course several natural extensions of our 
work that will undoubtedly generalize its applicabil- 
ity. Most notably, the linear mixing model of Equa- 




Figure 9: Along the top row are a pair of images of 
Renoir's On the Terrace with a reflection of Sheila pho- 
tographed through a linear polarizer at orthogonal ori- 
entations. Along the bottom row are the independent 
components. Also shown are the intermediate steps 
leading to the separation of the independent compo- 
nents. The third column shows the normalized joint 
histogram of the pair of images to its left. 
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Figure 10: Along the top row are a pair of images of 
Renoir 's Lunching on the Boating Party with a reflection 
of Sheila photographed through a linear polarizer at 
orthogonal orientations. Along the bottom row are 
the independent components. Also shown are the 
intermediate steps leading to the separation of the in- 
dependent components. The third column shows the 
normalized joint histogram of the pair of images to its 
left. 
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Figure 11: Shown along the top row are the pair of 
images photographed through a linear polarizer at 
orthogonal orientations, and shown along the bottom 
row are the separated components. 



tion (3) assumes a spatially uniform linear combination 
of source images that is unlikely to be true. The mixing 
will vary spatially because the polarization of reflected 
light depends on the angle to the camera, and this an- 
gle varies across the field of view. To account for this 
variation, our basic approach could be used in con- 
junction with a mixture model approach to fit multiple 
linear models [18]. Another possible extension would 
be to employ a more generic version of ICA that allows 
for a redundant system with more measurements than 
unknowns (e.g., [19]). In our case, this would mean 
photographing a scene at multiple polarization angles. 
Note though that since the polarization of a scene is 
fully characterized by three distinct orientations, there 
will be a point of diminishing returns. And lastly, our 
separation method could be applied selectively, for ex- 
ample when photographing a scene that is not com- 
pletely behind glass. In this case, the pair of images can 
be subtracted, and only regions with significant differ- 
ences will be considered to have a reflected component. 
This technique may also be useful in other domains. 



For example, reflections from the windshield of an au- 
tonomous vehicle with inboard cameras may cause prob- 
lems for visually guided navigation systems. Another 
possible application is in the field of surveillance where 
activities behind a reflective window may be revealed. 
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Appendix A 

The second statistical moment (variance) at an arbitrary angle 8\ is given by: 
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= J2 MO cos((?i) + 2/ 2 (0 sin(^)] 2 

8 = 1 

= J2 ^(') cos2 (^i) + 2j/i(0j/ 2 (0 cos((?i) sin(^i) + j/ 2 2 (0 sin 2 ^). 



The angle that maximizes the variance can be determined by first differentiating this error function: 
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And finally converting into polar coordinates with yi[i) = r[i) cos(</>(i)) and 1/2(1) = r[i) sin(</>(i)) yields the maximal 
solution as in Equation (7): 
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Appendix B 

The fourth statistical moment at an arbitrary angle 9 2 is given by: 
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= ^ [j/i(i) cos(# 2 ) + 2/ 2 (0 an(0 2 )] 



AT 



]T 2/4(0 cos 4 (# 2 ) + 42/?(i)2/ 2 (0 cos 3 (^ 2 ) sin(# 2 ) + 6y\(i)y 2 2 (i) cos 2 (# 2 ) sin 2 (# 2 ) 



+ 4j/i(i)i£(») cos(# 2 ) sin 3 (# 2 ) + y|(i) sin 4 (# 2 ) 



jY 



= E g2/i(0(3 + 4cos(2^ 2 ) + cos(4# 2 )) + j/f(i)2/ 2 (i)(sin(2^ 2 ) + I s in(4# 2 )) + \y\(i)yi(i)(\ - icos(4# 2 )) 

8 = 1 

+ 2/i(02/f(0(sin(2^ 2 ) - \ sin(4# 2 )) + |j/?(i)(3 - 4cos(2# 2 ) + cos(4# 2 )). (23) 

The angle that maximizes the fourth moment can be determined by differentiating this error function: 

^T^ = E g2/i(0(-8sin(2^ 2 ) - 4sin(4# 2 )) + 2/ 3 (02/2(0(2cos(2# 2 ) + 2cos(4# 2 )) + |j/ 2 (i)j/ 2 2 (i)(2sin(4^ 2 )) 



+ 2/i(02/ 2 (0(2cos(2^ 2 ) - 2cos(4# 2 )) + \y\ (i)(8sin(20 2 ) - 4sin(4# 2 )) 



(24) 



Note that this error function cannot be solved analytically but that the following normalized error function does lend 
itself to an analytic solution: 



E'(e 2 ) 

and 

dE'(9 2 ) 



||2/iH 2 +ll2/2|| 2 ^ 2) 



(25) 



1 



dE(6 2 ) 



IMP + IMI 2 de 2 

N 



1 



£ 7, (i\2 1 - 77x2 [(^ 4 (0 - 2/f (0) sin(2tf 2 ) + (2j/ 1 (i)j/ 2 3 (i) + 2y\(i)y 2 (i)) cos(ie 2 )} 

8 = 1 ^V / "^ 2/2 W 

1 F/ 

+ / -\2 1 / -\2 K 2 yiw 2 ; 
2/i(0+2/2(0 



[(-|2/i(0 " |2/ 2 (0 +32/i(02/f(0)sin(4^ 2 ) + (2s/f (i)sfit(i) - 2j/i(i)i|(i))cos(40 2 )] 



E 



j 2/i(0 2 + 2/2(0 : 



[(-52/i(0 - 5J/2 4 (0 + 32/ 2 (i)2/|(i))sin(4^ 2 ) + (2y\(i)yi{i) - 2j/ 1 (i)2/ 3 (i))cos(4^ 2 )] (26) 



where the 26* 2 terms cancel from the earlier maximization (see Appendix A) . This error function can now be maximized 
by setting equal to zero and solving: 



dE'(6 2 ) 



sin(46> 2 ) 







-E,-=i y^ly^ iZyK^yiji) - 2yi(»)^(»)) 

cos(4^ 2 ) - Ef=i , l( 8)4 2 (8^ -^i(0 - 2-2/ 2 4 (0 + 32/f(02/ 2 2 (0) 



1 

- tan 
4 



-N 



Ely 
8 = 1 ~f Y 



1 



y\(i) 2 +yi(i) 



■(2j/ 3 (02/ 2 (0-22/i(02/2 3 (0) 



^fcl yi(.-)4y2(i) 2 ("2^1(0 " 3^(0 + 32/ 2 (02/ 2 (0) 



(27) 
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And converting into polar coordinates with j/i (i) = r(i) cos((f)(i)) and 1/2(1) = r(i) sin((f)(i)) yields the maximal solution 
as in Equation (14): 



1 
-tan 



^N 2 



E:V 2 (0sin(4«K0) 



^N 



Er =1 ^ 2 (ocos(4^o) 



(28) 
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